Mobile device and method

ABSTRACT

A computer device comprises a display and a processor. The processor is configured to: intercept information sent from an application, to be displayed on the display; identify an information entity in the intercepted information, comprised in a predefined set of information entity types; reformat appearance of the identified information entity to be displayed on the display; forward the intercepted information including the reformatted appearance of the identified information entity to be displayed on the display. The display is configured to: present the identified information entity with the reformatted appearance to the user of the computer device. Also a corresponding method is disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/943,144 filed on Apr. 2, 2018, which is a continuation ofInternational Application No. PCT/CN2016/104742, filed on Nov. 4, 2016,which claims priority to European Patent Application No. EP15196779.1,filed on Nov. 27, 2015. All of the afore-mentioned patent applicationsare hereby incorporated by reference in their entireties.

TECHNICAL FIELD

Implementations described herein generally relate to a mobile device anda method therein. In particular is herein described a mechanism forenhancing user experience, based on user generated data from anapplication running in the mobile device.

BACKGROUND

Activity-awareness in mobile computing has inspired novel approaches forcreating smart personalized services/functionalities in mobile devices.The ultimate goal of such approaches is to enrich the user's experienceand enhance software functionality.

One of the major challenges in integrating mobile operating systems withactivity aware computing, is the difficulty of capturing and analyzingusers generated content from their personal handsets/mobile devices,without outweigh their privacy and securing the collected sensitivedata.

Although conventional solutions exist for collecting and extractingtextual contents generated by users in social computing applications,these solutions are most unsatisfying when it comes to personalintegrity of the user. All previously known conventional solutionscomprises collecting the user's generated content from variousapplications such as e.g. an email client and/or Short Message Service(SMS) applications; and more in general collect statistics about themobile device usage, related e.g. to visited web pages, communicatedparts, etc.

Such approach could provide a user with e.g. a reminder about theirflight, nearby places that match their preferences and may also enhancea search query performed by the user using his/her handset in aproactive manner. But unfortunately, all of those applications areintroduced to the user after exposing and sharing his/her personal datato a web services located outside the mobile device, e.g. in the cloud.In addition, the collected information is stored outside the user'spersonal mobile device in some remote server.

To extract personal/private information from a user's mobile device andsend it over a network to a central server for analysis and storage isdiscouraging to many users, who are concerned with their privacy.Further, extracted personal/private information may be exploited by amalicious third party (e.g. a burglar may detect that a certain userwill be on travel during a period, and use this information for timinghis break-in; or business critical information may leak to acompetitor).

These serious drawbacks make many users reluctant to use the describedconventional solutions. However, there is still a request forpersonalized pro-active services functionalities.

SUMMARY

It is therefore an object to obviate at least some of the abovementioned disadvantages and to enhance user experience, based on usergenerated data from an application running in a mobile device. This andother objects are achieved by the features of the appended independentclaims. Further implementation forms are apparent from the dependentclaims, the description and the figures.

According to a first aspect, a mobile device is provided. The mobiledevice comprises a display and a processor. The processor is configuredto intercept information sent from an application, to be displayed onthe display of the mobile device. Further the processor is alsoconfigured to identify an information entity in the interceptedinformation, comprised in a predefined set of information entity types.Also, the processor is further configured to reformat appearance of theidentified information entity to be displayed on the display. Inaddition, the processor is further configured to forward the interceptedinformation including the reformatted appearance of the identifiedinformation entity to be displayed on the display. The display isconfigured to present the identified information entity with thereformatted appearance to the user of the mobile device.

Thanks to the described aspect, information, such as display commandssent from a third part arty application to a display applicationcomprising text, graphics, format, position, etc., which belongs to apredefined information entity type may be detected by the processor. Thepredefined information entity type may e.g. be related to time and placefor a meeting, as inputted in a communication exchange in the thirdparty program by a user of the mobile device or his/her communicationcounterpart. Such detected information may be reformatted beforepresentation on the display, e.g. by high lighting certain information,providing an internet shortcut or insert meeting data into a calendarprogram of the mobile device, etc. Thus user experience is enhanced.

By performing the method locally in the mobile device, privacy of theuser is not breached. Neither the structure of the text is breached bythe provided solution.

In a first possible implementation of the mobile device according to thefirst aspect, wherein the application is an instant messagingapplication and the intercepted information comprises an instantmessage, the processor is configured to determine a display location ofthe instant message on the display, based on the interceptedinformation. Further the processor is additionally configured todetermine a source of origin of the instant message based on thedetermined display location. The processor is also configured toreformat the appearance of the identified information entity based onthe determined source of origin.

Thereby, by determining the display location of intercepted information,the source of origin of the information, i.e. the user of the mobiledevice, or the communication counterpart of the user, may be determined.By reformatting the information based on the source of origin, userexperience may be further enhanced.

In a second possible implementation of the mobile device according tothe first aspect, or the first possible implementation of the mobiledevice according to the first aspect, the processor is furtherconfigured to map the identified information entity with a furtherapplication, based on the information entity type of the identifiedinformation. The processor is also configured to create a shortcut linkbetween the identified information entity and the mapped application.Further the display of the mobile device is configured to display theshortcut link.

Thereby, further implementation is specified, enhancing user experienceand simplifying mobile usage of the user.

In a third possible implementation of the mobile device according to thefirst aspect, or any of the earlier described possible implementationsthereof, the intercepted information is a control message sent from theapplication and addressed to a display application. The control messagecomprising at least the information entity to be displayed on thedisplay and indicating an appearance for displaying the informationentity on the display and preferably a location for displaying theinformation entity on the display. The processor is configured toforward the intercepted information including the reformatted appearanceof the identified information entity to the display application.

By intercepting control commands sent from a (3rd) party application toa display application (configured to control the display of the mobiledevice), the reformatting of the information entity can be performedwithout having access to the 3^(rd) party application or withoutaltering a source code of the 3^(rd) party application. Hence, for the3^(rd) party application the complete process of reformatting isinvisible. Therefore it is enabled that, independent of the type andcapabilities of the 3^(rd) party application user experience can beenhanced by adding the functionalities as described in the presentdisclosure.

In a fourth possible implementation of the mobile device according tothe first aspect, or any of the earlier described possibleimplementations thereof, the processor is further configured to identifyinformation entity types comprising any of: person name, location, time,date, address, URL, phone number and/or email address presented as text,audio or image.

Thereby, the information entity types are further specified. Bydetecting and extracting these information entity types from aconversation on the mobile device (entered by the user or by the otherpart), an appointment concerning e.g. a flight, business meeting ordoctor appointment may be detected and stored in a calendar applicationand a reminder may be set to remind the user of the appointment. Thususer experience is increased and mobile device usage is simplified.

In a fifth possible implementation of the mobile device according to thefirst aspect, or any of the earlier described possible implementationsthereof, the processor is further configured to denote an event in theuser's calendar application at the identified time/date according to theinformation entity, when an information entity is identified as a timerelated information entity type.

By recognizing information calendar related information, e.g. in a chattext or message exchange, such recognized information may be inserted inthe user's calendar program. Thereby the user is relieved from manualentrance of meeting data into the calendar program, which is time savingand convenient for the user. Also, the risk of mistyping data into thecalendar program is reduced, or even omitted. Thereby the risk ofmissing a meeting, appointment or scheduled event is reduced.

In a sixth possible implementation of the mobile device according to thefirst aspect, or any of the earlier described possible implementationsthereof, the processor is further configured to map the identifiedinformation entity and the application based on the information entitytype, by mapping: a person name to a phone book application; a locationto a map application; a time/date to a calendar application; an addressto a map application; an URL to a web browser application; a phonenumber to an SMS application; and/or an email address to an emailingapplication.

Thereby, further implementation is specified, enhancing user experienceand simplifying mobile usage of the user.

In a seventh possible implementation of the mobile device according tothe first aspect, or any of the earlier described possibleimplementations thereof, the processor is further configured to reformatappearance of the identified information entity in the application byrich text formatting when the information entity comprises text.

Thereby essential information in a text message may be immediatelyrecognized and read by the user, thereby further enhancing userexperience.

In an eighth possible implementation of the mobile device according tothe first aspect, or any of the earlier described possibleimplementations thereof, the processor is further configured to set areminder, reminding the user of an event denoted in the user's calendarapplication.

Thereby, the user may be notified about the detected event, assistingthe user in avoiding missing the detected event.

In a ninth possible implementation of the mobile device according to thefirst aspect, or any of the earlier described possible implementationsthereof, the processor is further configured to add a clickable event,linking to a calendar application, showing detailed information aboutthe meeting.

Thus the user could easily find the event in the calendar applicationand e.g. check that the inserted data is correct and wherein the usermay enter additional information into the calendar application.

According to a second aspect, a method performed locally in a mobiledevice is provided. The method comprises intercepting information sentfrom an application, to be displayed on a display of the mobile device.Further the method comprises identifying an information entity in theintercepted information, comprised in a predefined set of informationentity types. In addition, the method also comprises reformattingappearance of the identified information entity to be displayed on thedisplay. The method also comprises forwarding the interceptedinformation including the reformatted appearance of the identifiedinformation entity to be displayed on the display. Additionally, themethod further comprises presenting the identified information entitywith the reformatted appearance to the user of the mobile device.

Thanks to the described aspect, information, such as display commandscomprising text, graphics, format, position, etc., which belongs to apredefined information entity type may be detected from a third partyprogram. The predefined information entity type may e.g. be related totime and place for a meeting, as inputted in a communication exchange inthe third party program by a user of the mobile device or his/hercommunication counterpart. Such detected information may be reformattedbefore presentation on the display, e.g. by high lighting certaininformation, providing an interne shortcut or insert meeting data into acalendar program of the mobile device, etc. Thus user experience isenhanced.

By performing the method locally in the mobile device, privacy of theuser is not breached. Neither the structure of the text is breached bythe provided solution.

In a first possible implementation of the method according to the secondaspect, wherein the application is an instant messaging application andthe intercepted information comprises an instant message, the methodfurther comprises determining a display location of the instant messageon the display, based on the intercepted information. Further the methodcomprises determining a source of origin of the instant message based onthe determined display location. Additionally, the method also comprisesreformatting the appearance of the identified information entity basedon the determined source of origin.

Thereby, by determining the display location of intercepted information,the source of origin of the information, i.e. the user of the mobiledevice, or the communication counterpart of the user, may be determined.By reformatting the information based on the source of origin, userexperience may be further enhanced.

In a second possible implementation of the method according to thesecond aspect, or the first possible implementation of the secondaspect, the method also comprises mapping the identified informationentity with a further application, based on the information entity typeof the identified information. The method furthermore comprises creatinga shortcut link between the identified information entity and the mappedapplication.

Thereby, further implementation is specified, enhancing user experienceand simplifying mobile usage of the user.

In a third possible implementation of the method according to the secondaspect, or any of the earlier described possible implementationsthereof, the intercepted information is a control message sent from theapplication and addressed to a display application. The control messagecomprising at least the information entity to be displayed on thedisplay and indicating an appearance for displaying the informationentity on the display and preferably a location for displaying theinformation entity on the display. The method also comprises forwardingthe intercepted information including the reformatted appearance of theidentified information entity to the display application.

Implementation details of the mobile device method are thereby furtherspecified.

In a fourth possible implementation of the method according to thesecond aspect, or any of the earlier described possible implementationsthereof, the information entity types comprise any of: person name,location, time, date, address, URL, phone number and/or email addresspresented as text, audio or image.

Thereby, the information entity types are further specified. Bydetecting and extracting these information entity types from aconversation on the mobile device (entered by the user or by the otherpart), an appointment concerning e.g. a flight, business meeting ordoctor appointment may be detected and stored in a calendar applicationand a reminder may be set to remind the user of the appointment. Thususer experience is increased and mobile device usage is simplified.

In a fifth possible implementation of the method according to the secondaspect, or any of the earlier described possible implementationsthereof, the method also comprises denoting an event in the user'scalendar application at the identified time/date according to theinformation entity when an information entity is identified as a timerelated information entity type.

By recognizing information calendar related information, e.g. in a chattext or message exchange, such recognized information may be inserted inthe user's calendar program. Thereby the user is relieved from manualentrance of meeting data into the calendar program, which is time savingand convenient for the user. Also, the risk of mistyping data into thecalendar program is reduced, or even omitted. Thereby the risk ofmissing a meeting, appointment or scheduled event is reduced.

In a sixth possible implementation of the method according to the secondaspect, or any of the earlier described possible implementationsthereof, the mapping between an identified information entity and theapplication is made, based on the information entity type, by mapping: aperson name to a phone book application; a location to a mapapplication; a time/date to a calendar application; an address to a mapapplication; an URL to a web browser application; a phone number to anSMS application; and/or an email address to an emailing application.

Thereby, the information entity types are further specified. Bydetecting and extracting these information entity types from aconversation on the mobile device, an appointment concerning e.g. aflight, business meeting or doctor appointment may be detected andstored in a calendar application and a reminder may be set to remind theuser of the appointment. Thus user experience is increased and mobiledevice usage is simplified.

In a seventh possible implementation of the method according to thesecond aspect, or any of the earlier described possible implementationsthereof, the method further comprises reformatting appearance of theidentified information entity in the application by rich text formattingwhen the information entity comprises text.

Thereby essential information in a text message may be immediatelyrecognized and read by the user, thereby further enhancing userexperience.

In an eighth possible implementation of the method according to thesecond aspect, or any of the earlier described possible implementationsthereof, the method further comprises setting a reminder, reminding theuser of an event denoted in the user's calendar application.

Thereby, the user may be notified about the detected event, assistingthe user in avoiding missing the detected event.

In a ninth possible implementation of the method according to the secondaspect, or any of the earlier described possible implementationsthereof, the method further comprises adding a clickable event, linkingto a calendar application, showing detailed information about themeeting.

Thus the user could easily find the event in the calendar applicationand e.g. check that the inserted data is correct and wherein the usermay enter additional information into the calendar application.

According to a third aspect, a computer program with a program code forperforming a method according to the second aspect, or any possibleimplementation thereof, when the computer program runs on a computer.

The advantages of the computer program according to the third aspect arethe same as those for the corresponding device claims according to thefirst aspect and/or the method according to the second aspect. Therebyuser experience is enhanced, based on user generated data extracted froman application in the mobile device.

Other objects, advantages and novel features of the aspects of theinvention will become apparent from the following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments are described in more detail with reference toattached drawings, illustrating examples of embodiments of the inventionin which:

FIG. 1 is a block diagram illustrating a mobile device and a chatprogram conversation according to an embodiment.

FIG. 2 is a flow chart illustrating a parse and edit view hierarchyaccording to an embodiment.

FIG. 3 is a block diagram illustrating a text View Centre according toan embodiment.

FIG. 4 is a block diagram illustrating a View Centre according to anembodiment.

FIG. 5 is a flow chart illustrating data flow of clustering text viewbased on location, according to an embodiment.

FIG. 6 is a flow chart illustrating text formatting according to anembodiment.

FIG. 7A is a block diagram illustrating a mobile device and a chatprogram conversation according to an embodiment.

FIG. 7B is a block diagram illustrating a mobile device and a chatprogram conversation according to an embodiment.

FIG. 7C is a block diagram illustrating a mobile device and a chatprogram conversation according to an embodiment.

FIG. 8 is a flow chart illustrating a method in a mobile deviceaccording to an embodiment of the invention.

FIG. 9 is a block diagram illustrating a mobile device according to anembodiment of the invention.

DETAILED DESCRIPTION

Embodiments of the invention described herein are defined as a mobiledevice and a method performed locally in a mobile device, which may beput into practice in the embodiments described below. These embodimentsmay, however, be exemplified and realized in many different forms andare not to be limited to the examples set forth herein; rather, theseillustrative examples of embodiments are provided so that thisdisclosure will be thorough and complete.

Still other objects and features may become apparent from the followingdetailed description, considered in conjunction with the accompanyingdrawings. It is to be understood, however, that the drawings aredesigned solely for purposes of illustration and not as a definition ofthe limits of the herein disclosed embodiments, for which reference isto be made to the appended claims. Further, the drawings are notnecessarily drawn to scale and, unless otherwise indicated, they aremerely intended to conceptually illustrate the structures and proceduresdescribed herein.

FIG. 1 is a schematic illustration over a mobile device 100 and aconversation in a chat program environment.

The mobile device 100 may comprise a cellular telephone handset, asmartphone, a tablet, a mobile computer, a netbook, a laptop, a pair ofsmart glasses, a smartwatch, a gaming device, etc.

By studying how mobile users are using their mobile devices 100 forperforming their daily life routines it becomes possible to build a verypersonalized services/applications that can meet the user's requirementsand needs. The herein described method comprises a generic method forextracting textual content from 3rd party applications in the mobiledevice 100. The extracted and collected text may be analyzed using aclient-side Named Entity Recognition (NER) service for adding textformat and clickable events into the original text in an ambient way.

Various embodiments described herein contributes to the fields of smartmobile operating system and activity awareness computing. Specifically,is herein described a real-time approach for extracting textual contentfound in the mobile device 100. After having extracted and collected thedata, all keywords and named entities are recognized with NER service.The result of the NER service will be used to visually annotate andformat the text inside the application in ambient manner. In addition,some clickable events may be injected into the text. Such clickableevents may be used to propose a new navigation model, the new clickableevent may comprise a pair of executable activity and a data set.Further, according to some embodiments, the user is provided withability to navigate directly to the app with a very fine-grained data.An example of those pairs could include {(Short Message Service (SMS)Application, Number://0443350640), (Call, ContactName://Mikko Terho),(View, url://www.huawei.com)}. In this sense, data refers to auser-generated content intercepted from the 3rd party applications fusedwith the data collected (autonomously) from the 3rd party applications.On the other hand, application executable refers to an end point ofapplication that can be used to run/execute a specific data format likephone number, Uniform Resource Locator (URL), Location, or Image, etc.Having such features inside any 3rd party application could enrich thequality of user's experience.

According to some embodiments, the activity based computing solution maybe able to first grasp the context, and then present data program pairsthat fit the occasion in real-time. Here the basic user interactionmodel is to present the most relevant data program pair for interactionin an editable work area. Data is an item known to the consumer such ase.g., calendar event, contact in phonebook, e-mail, short message or URLto a relevant web page. View is the visualization part of theModel/View/Controller (MVC) concept that renders the item into thescreen for interaction and editing. After completion of the interactionwith the data, the system may automatically provide contextually mostrelevant data and actions for consequent interaction.

The ultimate objective of the described behavioral awareness model is tocapture and intercept the user's activities invisibly running as ambientservice. This invisible service is silently collecting the data withoutinterfering or interrupting the user on-going tasks. Monitoring anddetecting the textual content from 3rd party application in the mobiledevice 100, may gather all available information directly from themessaging loop of the mobile Operating System (OS), such as controlmessages comprising such information sent from the 3^(rd) partyapplication to a display application. This comprises information thatcan address the status and the action performed by the user on themobile device 100 and the user's engagement with 3rd party applicationslike engagement in a conversation in social applications, posting onsocial media applications, sharing images/data, editing a contact card,calling a friend, sending SMS, or sending email, and similarapplications for communication.

Some embodiments described herein contributes to the fields of smartmobile operating system and activity awareness computing. Specifically,a real-time approach for replacing the standard navigation model foundin a mobile device 100 is presented. When the user receives a meetingrequest from a friend via a communication application such as e.g.Instant Messaging (IM) application, this text can be accessed andanalyzed before it is shown on the display of the device. All mentionedNamed Entities (such information entities of a predefined set of theinformation entities) in the conversation may be identified. In the sameway, a new calendar event may be added to the calendar application inthe mobile device 100. The event will comprise information such aslocation, time, and name of the person(s) involved in the meeting. Inthe same way, using a runtime text injection, this portion of textinside the IM application may be highlighted, e.g. by underlining thetext or otherwise changing its visual appearance. In addition, aclickable event linked to the text portion may be added, so when theuser clicks on the text portion, it will take the user to calendarapplication to show a detailed information about the meeting, in someapplications. This functionality may be accomplished in ambient mannerwithout distracting the user functionality of the mobile device 100.

Using a runtime mechanism within the mobile device 100, text may beextracted from a 3rd party application in the mobile device 100. Textelements may be stripped from the application view by extending the OScapability. The OS may be Android in some embodiments, as furtherdiscussed and exemplified herein. However, other OS such as e.g. iOS,Windows Phone, Blackberry, Firefox OS, Sailfish OS, Tizen, Ubuntu Touch,Linux and/or Symbian may be utilized in some alternative embodiments.

In some embodiments, the Android Activity manager may be modified to runa view debugger that may extract the text from Text View Fields in theView.

By using a TextCatcher program, text may be extracted from the viewshown in FIG. 1 . The mechanism of extracting the textual informationmay require modifying and extending the Android Activity life cycle insuch a way it may call the TextCatcher program and pass to it the viewstructure. Android View may be structured as a Tree of root node callmRoot and all User Interface (UI) elements in the view are structured aschildren nodes. This is a general rule of construction, building anddrawing the UI elements in the view managed by the Android View Manager.To pass the View tree the Android Activity Manager may be extended topass the mRoot view, package name, and view name to the TextCatcherprogram as shown in FIG. 2 .

The TextCatcher class may, in some embodiments, be running inside theandroid.util package as part of the Android core framework. When theActivity is created in the mobile device 100 for the user, the“onResume( )” function in class Activity.Java pass the Package name,Activity name, and mRootView of the application to the TextCatcherclass. Normally, the Android Activity Manager call the function“reportLaunchTimeLocked( )” from ActivityRecord.Java to confirm that theview is completely displayed to the user. This function may be extendedin some embodiments, to call our “dumpView” function in the TextCatcherclass. The dumpView function parses the Tree structure of Android Viewby identifying all TextViews in the tree. Once a TextView is found, itstext content passed to TextCluster class, TextCluster implements aclustering algorithm to identify message content, sender name, receivername, and the usefulness of the text. The Cluster Algorithm isdemonstrated in FIG. 5 .

The algorithm may use a HashMap data structure to analyze the raw textand text attributes. The text may in some embodiments be clustered intothree HashMaps (left, right, Unknown) based on their appearance in theview. Thus the TextView is clustered based on the coordinates of theirposition in the view. Thereby the source of the message in the TextViewmay be identified.

Normally, in Instant Messaging (IM) applications, the received textappears in the left side of the View. The sender text appears in theright side of the view as shown in FIG. 1 . In the FIG. 1 , the receiveris Basel and the sender is the CA admin. Using the coordinates, it maybe identified weather the TextView should be cluster to the left clusteror right cluster based on its distance from the view centre.

Each mobile device model has a specific value of Density-independentPixel (DP), for example, Nexus 5, has 480 DPI. DP is a virtual pixelunit that is used when defining UI layout, to express layout dimensionsor position in a density-independent way. For each TextView the CentreDP value may be calculated, assuming the TextView has a rectangularshape as shown in FIG. 3 . Thus the position of the centre of theTextView rectangle is calculated/determined and compared with centrevalue of the parent view as shown in FIG. 4 .

The following equations are used to calculate the required values:dp=px/(Device_dp/160);  Eq 1X=(left+right)/2;  Eq 2Y=(top+bottom)/2;  Eq 3TextView_Center_DP=(X,Y);  Eq 4View_centre_DP=(ViewX,ViewY).  Eq 5

For example in Google Nexus 5: ViewX=480/160, ViewY=1060/160.

Density-independent pixel (DP): A virtual pixel unit that is used whendefining UI layout. This method may be used to express layout dimensionsor position in a density-independent way. The DP is equivalent to onephysical pixel on a 160 dpi screen, which is the baseline densityassumed by the system for a “medium” density screen. At runtime, thesystem transparently handles any scaling of the DP units, as necessary,based on the actual density of the screen in use. The conversion of DPunits to screen pixels is simple: px=DP*(dpi/160). For example, on a 240dpi screen, 1 DP equals 1.5 physical pixels. FIG. 5 provides an overviewof the algorithm used to cluster the Text View based on its locationagainst the centre of its root view.

Both clusters of Text Views are filtered to remove duplication andunnecessary words. After identifying a useful portion of text, theTextCluster passes the text to a Named Entity Service (NERS). The NERservice identifies if the text comprises Name, Location, Time andAddress, etc. The NER services enclosed named entities of text byspecial symbols; each symbol is used to identify a class of namedentities. For example, for each portion of text that has been identifiesas a time expression, the NER service add the Symbol “%%” before andafter the text. The NER services may use the function“surroundExactMatch(String text, String exact, String preTag, StringpostTag)”, for example, surroundExactMatch(text,“Finland”, “##”, ##”)indicated the substring Finland is a named entity for a location. EachNamed Entity (i.e. person name, address, location, URL, phone number,email, time and date) may be surrounded with a special character, eachspecial character is identifying the text format that can be applied tothis portion of text, in some embodiments.

The result of named entities may be returned to the TextCluster class.After receiving a list of Named Entities, the TextCluster may use“setSpanBetweenTokens( )” function to inject text formation andclickable events based on the special characters embedded in the text inthe previous step. The new formatting and the view sequence may bepassed to a function that able to edit the view node using the viewsequence id. After finding the right node, the new text may be injectedin it. The process of identifying a meaning full text and formatting itis shown in FIG. 6 , concerning Text formatting. The outcome fromanalyzing and formatting the text is demonstrated in the storyboardshown in FIG. 7A-7C.

Thus the method provides a mechanism for extracting, analyzing andformatting user's generated content from third party applications in amobile device 100, without having any information sent or treatedoutside the mobile device 100. In some embodiments, Android OperatingSystem View structure is used to reveal meaningful content. Further,textView attributes may be used to identify the location and source ofthe text. The text might come from the user of the mobile device 100, orsomeone who has sent this text to the user. In some embodiments, thecentre of the text view may be used to identify its location relativelyto the parent view. Further, a runtime text formatting may be made toinject text formatting at runtime so the text may be formatted based onbeing e.g. a name, location, time or address.

The Named Entity Service is running locally at the mobile device 100 toimprove the privacy of the user. The collected data is not sent orredirected to any external entity such as e.g. a cloud based service orserver.

The described embodiments may be used to collect a useful and meaningfulcontent and analysis by using a machine-learning algorithm. Having suchan algorithm, the mobile device 100 could predict the next pair of useraction data. Such prediction may be used to propose a new navigationmodel, comprising a pair of executable activity and a data set.

In some embodiments, the user may be provided with a list of applicationand data pairs, so the mobile user can navigate directly to the app witha very fine-grained data. An example of those application and data pairsmay comprise: {(SMS Application, Number://0443350640),(Call,ContactName://Mikko Terho), (View, url://www.huawei.com)}. In thiscontext, data refers to a user-generated content intercepted from themobile OS messaging loop, which also is fused with sensory datacollected autonomously from the mobile device 100. Applicationexecutable refers to an end point of application that can be used torun/execute a specific data format like phone number, URL, location, orimage.

The activity based computing solution according to embodiments disclosedherein may ultimately be able to firstly grasp the context, and thenpresent data program pairs that fit the occasion in real-time. The mostrelevant data program pair for interaction may be presented for the userin an editable work area on the display of the mobile device 100.

Data in the here used meaning may comprise an item known to the usersuch as e.g. calendar event, contact in phonebook, e-mail, short messageor URL to a relevant web page, etc. View is the visualization part ofthe Model/View/Controller (MVC) concept that renders the item into thedisplay for interaction and editing. After completion of the interactionwith the data, the system may automatically provide the contextuallymost relevant items and actions for consequent interaction.

In some of the described implementation embodiments, multiple items maybe presented for the user to select between. Every time the user choosesto navigate via our proposed prediction the system may improve its ownaccuracy as it learns from the choices the user is making, according tosome embodiments.

This type of computing environment, based on current state and predictednext state, may become data centric and moderate the current applicationmodel. In some embodiments e.g. in the widget example illustrated inFIGS. 7A-7C, a new prediction may already be computed and proposed. Thishome screen widget may predict a list of apps & data pairs to be used asshortcut for navigation, at certain time periods. Once the user selectsthe website icon it may navigate the user the stack overflow website.

FIG. 8 is a flow chart illustrating embodiments of a method 800performed locally in a mobile device 100 for enhancing user experience,based on user generated data extracted from an application running inthe mobile device 100.

The application may be a communication application such as SMS, e-mail,Instant Messaging (IM), or any other chat program or communicationprogram.

To appropriately perform the method 800 locally in a mobile device 100,without exporting any data out of the mobile device 100 and enhance userexperience, the method 800 may comprise a number of steps 801-810. It ishowever to be noted that any, some or all of the described step 801-810,may be performed in a somewhat different chronological order than theenumeration indicates, be performed simultaneously or even be performedin a completely reversed order according to different embodiments. Somemethod steps such as e.g. step 806-808 and 810 may be performed withinsome, but not necessarily all embodiments. Further, it is to be notedthat some steps may be performed in a plurality of alternative mannersaccording to different embodiments, and that some such alternativemanners may be performed only within some, but not necessarily allembodiments.

Any, some or all of step 801-810 may in some embodiments be periodicallyre-performed. The method 800 may comprise the following steps:

Step 801 comprises intercepting information sent from an application, tobe displayed on a display 940 of the mobile device 100.

The intercepted information may comprise user information input of theapplication.

The user information input may be made by the user of the mobile device100, or the communication partner of the user. The user informationinput may comprise text, drawings, images, graphics, format, position oraudio in different embodiments.

Step 802 comprises identifying an information entity in the interceptedinformation, comprised in a predefined set of information entity types.

Such predefined set of information entity types may comprise any of:person name, location, time, date, address, URL, phone number and/oremail address presented as text, audio or image.

Step 803 comprises reformatting appearance of the identified informationentity to be displayed on the display 940.

The reformatting of the information appearance may be made when theintercepted information is comprised in the predefined set ofinformation entity types, in some embodiments.

The reformatted appearance of the identified information entity may bebased on a determined source of origin, in some embodiments, i.e. theuser of the mobile device 100, or his/her communication partner.

The reformatted appearance of the identified information entity in theapplication may comprise e.g. rich text formatting in some embodiment,when the information entity comprises text.

Step 804 comprises forwarding the intercepted information including thereformatted appearance of the identified information entity to bedisplayed on the display 940.

However, in further embodiments some further steps may be performedbefore the intercepted information with the reformatted appearance isforwarded to display. As an example, a short cut link may be added orsome further analysis of the information entity can be performed, asdescribed in the following steps. It should be emphasized that thefollowing steps could also be performed before step 803 or as part ofstep 803.

Step 805, which may be performed only in some embodiments, may comprisedetermining a display location of the instant message on the display940, based on the intercepted 801 information, when the application isan instant messaging application and the intercepted 801 informationcomprises an instant message.

Instant Messaging (IM) is a set of communication technologies used forreal-time text-based communication between two or more participants overthe Internet or other types of networks. Short messages are typicallytransmitted bi-directionally between two parties, when each user choosesto complete a thought and select “send”. Some instant messagesapplications can use push technology to provide real-time text, whichtransmits messages character by character, as they are composed. In someembodiments, instant messaging can add e.g. file transfer, clickablehyperlinks, Voice over IP (VoIP), and/or video chat.

Further, in some embodiments, instant messaging may be made also whenthe receiver is off line.

Step 806, which only may be performed in some embodiments wherein step805 has been performed, may comprise determining a source of origin ofthe instant message based on the determined 805 display location.

The source of origin of the instant message may be the user of themobile device 100, or the communication partner of the user. The userinformation input may comprise text, drawings, images, graphics, format,position and/or audio, or any combination thereof, in differentembodiments.

Step 807, which may be performed only in some embodiments, may comprisemapping the identified 802 information entity with a furtherapplication, based on the information entity type of the identified 802information.

The mapping between an identified 802 information entity and theapplication may be made based on the information entity type, bymapping: a person name to a phone book application; a location to a mapapplication; a time/date to a calendar application; an address to a mapapplication; an URL to a web browser application; a phone number to anSMS application; and/or an email address to an emailing application, insome embodiments.

Step 808, which only may be performed in some embodiments wherein step807 has been performed, may comprise creating a shortcut link betweenthe identified 802 information entity and the mapped 807 application.

The shortcut link may be a handle in a user interface that allows theuser to find a file or resource located in a different directory orfolder from the place where the shortcut link is located. The shortcutlink may be implemented as a small file comprising a target UniformResource Identifier (URI) to an object, or the name of a target programfile that the shortcut represents.

The shortcut link may be touchable so that the user is enabled to accessthe mapped 807 application via the shortcut link when touching it on atouch display of the mobile device 100.

In some embodiments, the shortcut link may be linking to a calendarapplication, showing detailed information about the meeting. Theshortcut link may be a clickable event comprising a highlighted link insome embodiments, or e.g. a recognizable logotype.

Step 809 comprises presenting the identified 802 information entity withthe reformatted 803 appearance to the user of the mobile device 100.

Step 810, which may be performed only in some embodiments, may comprisedenoting an event in the user's calendar application at the identifiedtime/date according to the information entity, when an informationentity is identified 802 as a time related information entity type.

Further, in some embodiments, the user of an event denoted in the user'scalendar application may be reminded by setting an alarm. The optionalalarm (based on a sound, a light, a tactile signal or a combination ofthese) may be set a (brief) period in advance to the event in thecalendar application.

FIG. 9 illustrates an embodiment of a mobile device 100 for enhancinguser experience, based on user generated data from an applicationrunning in the mobile device 100. The mobile device 100 may perform atleast some of the above described method steps 801-810.

The mobile device 100 comprises a processor 920, configured to interceptinformation sent from an application, to be displayed on the display940, e.g. via a software module responsible for displaying the content.The information may comprise e.g. a display command comprising text,graphics, format, position, etc. The processor 920 is also configured toidentify an information entity in the intercepted information, comprisedin a predefined set of information entity types. Further the processor920 is further configured to reformat appearance of the identifiedinformation entity to be displayed on the display 940, e.g. when theinformation entity is comprised in the predefined set of informationentity types. Further the processor 920 is additionally configured toforward the intercepted information including the reformatted appearanceof the identified information entity to be displayed on the display 940,e.g. by a software module responsible for displaying the informationcontent.

The mobile device 100 also comprises a display 940 configured to presentthe identified information entity with the reformatted appearance to theuser of the mobile device 100.

The processor 920 may in some embodiments be configured to determine adisplay location of the instant message on the display 940, based on theintercepted information. The processor 920 may furthermore be configuredto determine a source of origin of the instant message based on thedetermined display location. Also, the processor 920 may be configuredto reformat the appearance of the identified information entity based onthe determined source of origin, when the application is an instantmessaging application and the intercepted information comprises aninstant message.

The processor 920 may in some further embodiments be configured to mapthe identified information entity with a further application, based onthe information entity type of the identified information. Further, theprocessor 920 may be additionally configured to create a shortcut linkbetween the identified information entity and the mapped application.The shortcut link may comprise a transfer of the identified informationentity to the other mapped application, in some embodiments.

Further, the display 940 may be configured to display the shortcut linkin some embodiments.

The processor 920 may in some embodiments be configured to add ashortcut link, linking to a calendar application, showing detailedinformation about e.g. a meeting, in case the information indicates ameeting is going to take place.

In some embodiments, wherein the intercepted information is a controlmessage sent from the application and addressed to a displayapplication, the control message comprising at least the informationentity to be displayed on the display 940 and indicating an appearancefor displaying the information entity on the display 940 and preferablya location for displaying the information entity on the display 940, theprocessor 920 may be configured to forward the intercepted informationincluding the reformatted appearance of the identified informationentity to the display application.

The processor 920 may in some embodiments be configured to identifyinformation entity types comprising any of e.g.: person name, location,time, date, address, URL, phone number and/or email address presented astext, audio and/or image.

In some embodiments, the processor 920 may be further configured todenote an event in the user's calendar application at the identifiedtime/date according to the information entity, when an informationentity is identified as a time related information entity type.

The processor 920 may also be further configured to map the identifiedinformation entity and the application based on the information entitytype, by mapping: a person name to a phone book application; a locationto a map application; a time/date to a calendar application; an addressto a map application; an URL to a web browser application; a phonenumber to an SMS application; and/or an email address to an emailingapplication.

Further, the processor 920 may be further configured to reformatappearance of the identified information entity in the application byrich text formatting when the information entity comprises text, in someembodiments.

In some embodiments, the processor 920 may in addition be configured toset a reminder, reminding the user of an event denoted in the user'scalendar application.

Such processor 920 may comprise one or more instances of a processingcircuit, i.e. a Central Processing Unit (CPU), a processing unit, aprocessing circuit, a processor, an Application Specific IntegratedCircuit (ASIC), a microprocessor, or other processing logic that mayinterpret and execute instructions. The herein utilized expression“processor” may thus represent a processing circuitry comprising aplurality of processing circuits, such as, e.g., any, some or all of theones enumerated above.

Furthermore, the mobile device 100 also may comprise a receiving circuit910 in some embodiments, for receiving wireless signaling.

The mobile device 100 also may comprise a transmitting circuit 930,configured to transmit wireless signals.

Furthermore, the mobile device 100 in addition may comprise at least onememory 925, according to some embodiments. The optional memory 925 maycomprise a physical device utilized to store data or programs, i.e.,sequences of instructions, on a temporary or permanent basis. Accordingto some embodiments, the memory 925 may comprise integrated circuitscomprising silicon-based transistors. Further, the memory 925 may bevolatile or non-volatile.

The method 800 comprising at least some of the method steps 801-810 maybe implemented through the one or more processors 920 in the mobiledevice 100 together with computer program product for performing thefunctions of the method 800.

Thus a computer program comprising program code for performing themethod 800 according to any embodiment of method steps 801-810, may beperformed when the computer program is loaded in the processor 920.

The computer program product mentioned above may be provided forinstance in the form of a data carrier carrying computer program codefor performing at least some of the method steps 801-810 according tosome embodiments when being loaded into the processor 920. The datacarrier may be, e.g., a hard disk, a CD ROM disc, a memory stick, anoptical storage device, a magnetic storage device or any otherappropriate medium such as a disk or tape that may hold machine readabledata in a non-transitory manner. The computer program product mayfurthermore be provided as computer program code on a server anddownloaded to the mobile device 100, e.g., over an Internet, WiFi orintranet connection.

The terminology used in the description of the embodiments asillustrated in the accompanying drawings is not intended to be limitingof the described method 800 and mobile device 100. Various changes,substitutions and/or alterations may be made, without departing from theinvention as defined by the appended claims.

As used herein, the term “and/or” comprises any and all combinations ofone or more of the associated listed items. The term “or” as usedherein, is to be interpreted as a mathematical OR, i.e., as an inclusivedisjunction; not as a mathematical exclusive OR (XOR), unless expresslystated otherwise. In addition, the singular forms “a”, “an” and “the”are to be interpreted as “at least one”, thus also possibly comprising aplurality of entities of the same kind, unless expressly statedotherwise. It will be further understood that the terms “includes”,“comprises”, “including” and/or “comprising”, specifies the presence ofstated features, actions, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, actions, integers, steps, operations, elements,components, and/or groups thereof. A single unit such as e.g. aprocessor may fulfil the functions of several items recited in theclaims. The mere fact that certain measures are recited in mutuallydifferent dependent claims does not indicate that a combination of thesemeasures cannot be used to advantage. A computer program may bestored/distributed on a suitable medium, such as an optical storagemedium or a solid-state medium supplied together with or as part ofother hardware, but may also be distributed in other forms such as viaInternet or other wired or wireless communication system.

The invention claimed is:
 1. A computer device, comprising: a display;at least one processor; and one or more memories coupled to the at leastone processor and storing programming instructions for execution by theat least one processor to cause the computer device to: interceptinformation to be displayed on the display, wherein the information issent through an application; identify, in the intercepted information,an information entity that matches at least one information entity typein a predefined set of information entity types, wherein the predefinedset of information entity types comprises person name, address, andphone number, and wherein each information entity type in the predefinedset of information entity types has a corresponding relationship with amapped application; reformat appearance of the identified informationentity to be displayed on the display, wherein the application is aninstant messaging application, the intercepted information comprises aninstant message when identifying the information entity that matches theat least one information entity type in the predefined set ofinformation entity types, and display the intercepted informationincluding the reformatted appearance of the identified informationentity on the display, wherein the reformatted appearance of theidentified information entity is different from an appearance of aremaining portion of the intercepted information; map the identifiedinformation entity with a mapped application, based on the informationentity type of the identified information entity and the correspondingrelationship; create a shortcut link between the identified informationentity and the mapped application, wherein the shortcut link causes themapped application to execute on the computer device when selected; anddisplay the shortcut link.
 2. The computer device according to claim 1,wherein the intercepted information is a control message sent from theapplication and addressed to a display application, the control messagecomprising at least the information entity to be displayed on thedisplay and indicating an appearance for displaying the informationentity on the display and a location for displaying the informationentity on the display; and wherein the programming instructions, whenexecuted by the at least one processor, cause the computer device toforward the intercepted information including the reformatted appearanceof the identified information entity to the display application.
 3. Thecomputer device according to claim 1, wherein the programminginstructions, when executed by the at least one processor, cause thecomputer device to change appearance of the identified informationentity in the application by rich text formatting when the informationentity comprises text.
 4. A displaying method, comprising: interceptinginformation sent through an application, to be displayed on a display ofa computer device; identifying, by the computer device, in theintercepted information, an information entity that matches at least oneinformation entity type in a predefined set of information entity types,wherein the predefined set of information entity types comprises personname, address, and phone number, and wherein each information entitytype in the predefined set of information entity types has acorresponding relationship with a mapped application; when identifyingthe information entity that matches the at least one information entitytype in the predefined set of information entity types, reformattingappearance of the identified information entity to be displayed on thedisplay, wherein the application is an instant messaging application,the intercepted information comprises an instant message, and displayingthe intercepted information including the reformatted appearance of theidentified information entity on the display, wherein the reformattedappearance of the identified information entity is different from anappearance of a remaining portion of the intercepted information;mapping the identified information entity with a mapped application,based on the information entity type of the identified informationentity and the corresponding relationship; creating a shortcut linkbetween the identified information entity and the mapped application,wherein the shortcut link causes the mapped application to execute whenselected; and displaying the shortcut link on the computer device. 5.The displaying method according to claim 4, wherein the interceptedinformation is a control message sent from the application and addressedto a display application, the control message comprising at least theinformation entity to be displayed on the display and indicating anappearance for displaying the information entity on the display and alocation for displaying the information entity on the display; andwherein the displaying method further comprises: forwarding theintercepted information including the reformatted appearance of theidentified information entity to the display application.
 6. Thedisplaying method according to claim 4, wherein the displaying methodfurther comprises: changing appearance of the identified informationentity in the application by rich text formatting when the informationentity comprises text.
 7. A non-transitory computer-readable mediumstoring a program code which, when executed by a computer, causes thecomputer to perform operations comprising: intercepting information sentthrough an application, to be displayed on a display of a computerdevice; identifying, by the computer device, in the interceptedinformation, an information entity that matches at least one informationentity type in a predefined set of information entity types, wherein thepredefined set of information entity types comprises person name,address, and phone number, and wherein each information entity type inthe predefined set of information entity types has a correspondingrelationship with a mapped application; when identifying the informationentity that matches the at least one information entity type in thepredefined set of information entity types, reformatting appearance ofthe identified information entity to be displayed on the display,wherein the application is an instant messaging application, theintercepted information comprises an instant message, and displaying theintercepted information including the reformatted appearance of theidentified information entity on the display, wherein the reformattedappearance of the identified information entity is different from anappearance of a remaining portion of the intercepted information;mapping the identified information entity with a mapped application,based on the information entity type of the identified informationentity and the corresponding relationship; creating a shortcut linkbetween the identified information entity and the mapped application,wherein the shortcut link causes the mapped application to execute whenselected; and displaying the shortcut link on the computer device. 8.The non-transitory computer-readable medium according to claim 7,wherein the intercepted information is a control message sent from theapplication and addressed to a display application, the control messagecomprising at least the information entity to be displayed on thedisplay and indicating an appearance for displaying the informationentity on the display and a location for displaying the informationentity on the display; and wherein the operations further comprise:forwarding the intercepted information including the reformattedappearance of the identified information entity to the displayapplication.
 9. The non-transitory computer-readable medium according toclaim 7, wherein operations further comprise: changing appearance of theidentified information entity in the application by rich text formattingwhen the information entity comprises text.